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Technology-mediated simulations of teaching practice are becoming a more common way to 
introduce teachers to the dilemmas of teaching during professional development. In this paper, 
we show that the inclusion of markers of student emotions in cartoon-based scenarios of 
teaching changes teachers’ appropriateness rating of the actions that the teacher took in the 
storyboard. Our results show that the inclusion of markers of student emotions in representations 
of practice could cue teachers into a particular judgments of action. 
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Computer-based virtual environments, using avatars to represent students, have grown in 
popularity in both teacher education and educational research (Gibson, 2007; Herbst, Chazan, 
Chen, Chieu, & Weiss 2011; Ma et al., 2016; Straub, Dieker, Hynes, & Hughes, 2014). While 
such environments have been used to engage teachers in the cognitive and rational aspects of 
teaching, they are not frequently used to elicit teachers’ reasoning related to more affective 
aspects of teaching (Lehtonen, Page, Miloseva, & Thorsteinsson, 2008). However, similar 
technology, using avatars in virtual environments, have been shown to be effective in helping 
individuals gain increased understandings of social situations (e.g. Cheng & Ye, 2010; Tettegah, 
2005). This suggests the potential for such environments for supporting teachers’ engagement 
with the more emotional aspects of teaching. 

This is good news, given that emotions arguably play an important role in teachers’ decision 
making (Demetriou, Wilson, & Winterbottom, 2009; Hargreaves, 1998; Zembylas, 2005). For 
some time now, we have known that emotional intelligence —the ability to perceive one’s own 
emotional state—has evident value in professional circumstances (e.g. Jaeger, 2003; Lopes et al., 
2006), including playing an important role in teacher’s job satisfaction and organizational 
commitment (Naderi Anari, 2012). Beyond the many studies on teacher’s emotional 
intelligence, there is reason to believe that the broader construct of emotional competence—the 
ability to recognize and understand the emotions of others—may be highly related to improved 
outcomes for student learning (Jennings & Greenburg, 2009). This hypothesis seems reasonable 
in light of recent findings that the relational aspects of teaching are as important as the cognitive 
aspects for student outcomes (Battey & Neal, 2018). Knowing how teachers’ emotional 
competence shapes their work is particularly important given they often have to know what 
people are feeling without having to directly probe them to report their feelings—e.g., reading 
students’ emotions when in the midst of teaching. However, learning to effectively read facial 
expressions is non-trivial and particularly challenging during instructional transactions where 
there exists a power differential between the individuals, as is the case with teachers and students 
(Galinsky, Gruenfeld, & Magee, 2003). Yet, little work in the field has aimed to gauge or 
support teachers in gaining such competencies (Hargreaves, 1998). 

We see the work we describe in this paper as adding to this literature about the role of 
emotions, in particular teachers’ emotional competence, in teaching. We have been designing 
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scenario-based assessments to understand teachers’ reasoning about mathematics instruction. In 
that work, we have been cautious about using facial expressions because of our own lack of 
certainty about how facial expressions might impact the ways in which participants’ judge the 
instructional actions represented within such items. More specifically, we have been uncertain 
whether the addition of facial expressions might bias the items for individuals more capable of 
reading facial expressions (in both human faces as well as within cartoon representations). In that 
context, we have been wondering whether and how emotions represented on cartoon-based 
representations of students in classroom scenarios of practice play a role in the ways that 
teachers appraise the scenarios of teaching. 


Theoretical Framework 

Digital simulations of teaching use a variety of semiotic resources to represent the 
individuals and settings of practice. Icons, language, and dashboards (e.g., in SimSchool, Gibson, 
2007) are three kinds of semiotic resources frequently used for students to respond to instruction, 
but for each of those a limited variety of systems of choice are available. Herbst et al. (2011) 
argue that graphic representations of teaching practice using two-dimensional, nondescript icons 
to represent teacher and students can display nonverbal aspects of teaching practice. They argue 
that comic-based representations can represent the same meaning as, for example, textual 
accounts of practice. Herbst and Chazan (2015) suggest that lean, nondescript graphic elements 
allow the designer to easily represent practice as emergent in mathematics classroom interactions 
and enable viewers to project into those characters their own settings and clients—in contrast 
with avatars whose graphical features are used to mark high individuality. In a follow-up study, 
Herbst, Boileau, Clark, Milewski, Chieu, Giirsel, and Chazan (2017) show that this lean cartoon- 
based semiotic system can be expanded to include signifiers for complex social constructs such 
as race and ethnicity. In their comparison of the written story and the storyboard for the Case of 
Mya (Chazan, Herbst, & Clark, 2016), Herbst et al. (2017) conclude that the storyboard 
representation encouraged a plurality of interpretations of the same case, both expected and 
unexpected. 

Following this line of inquiry, our lab has begun exploring whether emotions can be encoded 
using a similarly lean semiotic system when representing professional scenarios of teaching. One 
crucial foundation for that work is the well-established line of research from Ekman and 
colleagues who developed the facial action coding system (FACS) (Ekman & Friesen, 1978)— 
an analytic scheme for precisely describing various configurations of human faces as functions 
of muscle movements. The FACS has made possible the cataloging of human facial expressions 
for reliably representing human emotion interpretable by humans across cultural boundaries 
(Ekman & Friesen, 1986). The reliable interpretation of emotions based on facial expression 
representations suggests that the ability to interpret facial expressions is an important aspect of 
emotional intelligence (Keltner, Sauter, Tracy, Cowen, 2019). The FACS has also been used to 
inform the development of animated and cartoon facial expressions (McCloud, 2006; Thorisson, 
1996; Spindler & Fadrus, 2009). 

In this paper, we leverage the FACS to modify the default expressions of a set of cartoon 
characters we call the Thexpians B (Herbst & Chieu, 2011) to include lean semiotic markers of 
emotions (Dimmel, Milewski, & Herbst, 2015). For example, Figure 1 shows the affordances of 
the facial expression semiotic system to represent student emotions. In both panels of Figure 1, a 
student disagrees with a peer who has critiqued his shared solution. In the left panel, the 
student’s eyes and mouth express a neutral state. In the right panel, we added to the eyes 
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triangular-shaped eyebrows that slope inward and changed the shape of his mouth from round to 
triangular to represent the emotional state of anger. These changes are meant to communicate the 
emotions associated with that specific moment. 

In previous work, we have compared these kinds of lean renderings of facial expressions 
with that of facial expressions as represented by human actors and demonstrated that participants 
are, in fact, able to interpret the cartoon-based representations of emotion with similar levels of 
accuracy to that of photos (see Dimmel et al., 2015). Here, we build on that work to ask how the 
addition of such expressions into scenarios of teaching make a difference for participants’ 
judgment of the instructional actions taken within such scenarios. To that end, we ask: To what 
extent and how does the representation of emotions on student avatars in a cartoon-based 
representation of practice change the way that practicing teachers judge the appropriateness of 
instructional actions of teacher avatars? 


Neutral Character Emoted Character 


No. | want to do this first and divide by 


No. | want to do this first and divide by 
16. So it's going to be 120 by 120. 


16. So it's going to be 120 by 120. 


Manny and his dad are preparing for 
a day of yardwork. They are going to 
lay out pavers sized 4 inches by 4 
nches to cover 100 square feet. They 
30 to the store for boxes that have 

5 pavers each. How many boxes do 
they need to buy? 


Manny and his dad are preparing for 
a day of yardwork. They are going to 
lay out pavers sized 4 inches by 4 
inches to cover 100 square feet. They 
go to the store for boxes that have 

5 pavers each. How many boxes do 
they need to buy? 


120 x 12C 120 x 120 


| don't think so. | am 
multiplying 16 by 5. 


| don't think so. | am 
multiplying 16 by 5. 


> n/t 


The question is about how many 
* | boxes you need to get. 


The question is about how many 
boxes you need to get. 


= = a Cl 


Figure 1: Neutral and Emoted Characters 


Methods 

Participants 

In this study, we use a randomized control trial design to assess the extent to which 
representations of emotions impact teacher interpretations of scenarios of teaching. We recruited 
69 participants from the respondent pool for the validation study of the PROSE instrument 
(Herbst & Ko, 2018). We randomly assigned participants to either the emoted or neutral 
experimental conditions. For the emoted condition, we edited the original PROSE items to 
include relevant facial expressions at the scenario turning point (as shown on the right pane of 
Figure 1). For the neutral condition, we removed all facial expressions from the original PROSE 
items. Instead, we represented all characters in the scenario using the neutral eye and mouth 
shapes (as shown on the left pane of Figure 1). Intuitively, comparing the responses to the 
emoted and neutral scenarios allows us to assess the extent to which the explicit representation of 
facial expressions in scenarios of teaching has an impact on teachers’ interpretation of 
instructional actions. 
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Outcomes 

The main outcome for this study is teachers’ reactions to scenarios of instruction. To this 
end, we adapt the PROSE instrument (Herbst & Ko, 2018) to include facial expressions. The 
main outcome of interest of the unadapted PROSE instrument is teachers’ recognition of 
professional obligations as a source of justification of instructional actions. In that instrument, 
teachers are asked to assess the extent to which a teacher instructional action was justified on the 
grounds of professional obligations (Herbst & Chazan, 2012). In the adaptation of this 
instrument to our study, we selected nine items! from the overall battery of twenty-four PROSE 
items for two possible reasons. First, following Milewski and Erickson (2015), we located items 
where a sizable number of participants indicated that their rating of the item depended on some 
assumptions they needed to make about the scenarios—we call these high construal items. We 
scanned teachers’ responses to high construal items to locate items for which participants 
professed needing more information about the emotions of the student avatars—we call these 
high emotional construal items. For example, participants indicated their rating depended on 
whether or not a student in the scenario was upset. Accompanying these responses, we also 
found many participants that assumed that the represented student was in fact upset (even though 
the expression on the student’s face was neutral in the item). That said, not all responses 
contained references to emotion and presumably some participants did not construe emotion into 
the item and thus the inclusion of emotional cues into the item might alter the way that 
participants respond to the item. Using this heuristic, we located seven items for inclusion in the 
study. Second, some items in the original PROSE instrument included some facial expressions 
which participants could have interpreted as emotions. We call these items PROSE emoted items 
and for the purposes of this study selected two such items. For all of these items, either the data 
or the original design of the item led us to believe that emotion may have been part of the way 
that participants were reasoning about their judgments. Our main outcome of interest with these 
adapted items from the PROSE instrument was to understand more about how the inclusion of 
facial expressions as markers of emotion was affecting the ways in which participants’ rated the 
items. Figure 2 reports a brief synopsis of each item and whether it was selected for its high 
emotional construal or PROSE emoted items. 

Participants also completed an emotion recognition questionnaire (Keltner et al., 2019). This 
instrument shows participants photographs of trained actors displaying emoted facial expressions 
and asks participants to identify the corresponding emotion. We also administered a parallel 
version of this instrument (developed by Dimmel et al, 2015) that uses our cartoon characters— 
using cartoon-based shapes for eyes, eyebrows, and mouth as well as changing their location in 
relation of the overall face to reproduce the facial expressions in the photographs. These two 
instruments allowed us to measure the extent to which our participants are able to recognize 
emotions in both photographs and cartoons. We use this information as a robustness check of our 
results. 


Table 1: Description of Selected PROSE Items 


Item Description Type 
al012 A/n (embarrassed) student shares a solution to a problem on the board. High 
The teacher asks others to build on that solution strategy. Construal 


' One item was removed from these analyses because of a mistype in the question prompt. 
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a3012 A student is sharing her/his solution on the board. Another student in High 
the classroom suggests another solution strategy and the student at the — Construal 
board responds (with anger). The teacher compliments the students 
handling the disagreement. 

a3022 The teacher is asking questions to the class. One student answers all the High 
questions, while other (frustrated) students look on. The teacher calls Construal 
on students by pulling names from a cup. 

a3032 A (confused) student privately asks the teacher a question. The teacher High 


asks the student to ask the question out loud to the class. Construal 
a3102 A student tells the teacher that she/he is done with her work. The High 
teacher asks her/him to help other students instead of working on the Construal 
homework and she/he (begrudgingly) complies. 
a3112 A teacher ignores volunteering students to cold-call a (confused) High 
student to publicly share a definition of a term. Construal 


a3152 ~—_—A student is solving a problem on the board. The teacher asks her/him High 
to start over because other (frustrated) students cannot read the writing. Construal 


a3162 =‘ Teacher offers magnets to (excited) students. When many students Emoted 
volunteer, the teacher decides to raffle the magnets. PROSE 
Analysis 


Our data is nested both within respondent and item. That is, we asked the same participants 
to answer multiple items. As we are interested in the effect of facial expressions on the overall 
evaluation of the instructional action, we use a multilevel linear model to account for the nesting 
of responses within participants and items. In detail, we use a mixed effects model (Raudenbush 
& Bryk, 2002) with crossed participant and item random effects. We fit the equation 


Ypi = Po + Bi: Emoted,; + Up + uj + epi 


where U, is the participant-level random effect accounts for participant-level differences in the 
assessment of the instructional action that is unrelated to our experimental condition, u; is the 
item-level random effect accounts for item-specific differences in the assessment of the 
instructional action that are due to contextual factors embedded in the specific PROSE 
instrument. The coefficient of interest is 6,. This coefficient estimates the causal effect of 
including facial expressions (i.e., the difference between seeing an emoted item versus a neutral 
item) on participant assessment of the instructional action represented in the PROSE items. 


Results 

Effect of Emotions on the Assessment of Instructional Actions 

Table 1 reports the estimates of the effect of facial expressions on teachers’ assessment of 
instructional actions. Column | reports the results of the unconstrained mixed effects model, 
column 2 reports the effect on the raw assessment score (1-6 Likert scale), column 3 reports the 
same effect in standard deviation units. Columns 4 and 5 report the results of robustness checks 
testing whether our main results are sensitive to participants’ recognition of emotions in 
photographs or cartoons. 
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We find that the inclusion of facial expressions in scenarios of teaching has an impact on 
teachers’ assessment of instructional actions of, on average and in absolute value, 0.356 points 
on a 1-6 Likert scale or 0.280 standard deviation units. This result appears to be robust against 
participants’ recognition of emotions in photographs or cartoons. That is, the magnitude of the 
effect does not change after controlling for participants’ emotion recognition in photographs and 
cartoon images. This allows us to conclude that participants’ change in their assessment of the 
instructional actions in emoted items is due to the emotions that are represented in the characters 
facial expressions. If this was not the case, we would observe a change in the estimated effect 
once controlling for participants’ emotion recognition. 


Table 2: Average Effect of Facial Expressions on the Assessment of Instructional Actions 


(1) (2) (3) (4) (5) 
te STD + STD + 
Unconditional Pooled STD Gastaaniies Photo Rec 
Emoted 0.356" 0.260" 0.262" 0.267°" 
(0.111) (0.081) (0.082) (0.081) 
Constant 1.798°" 1.5837" 0.000" 0.022 0.121 
(0.239) (0.247) (0.063) (0.185) (0.213) 
Var(Resp) 0.014 0.000 0.000 0.000 0.000 
Var(Item) 0.430 0.430 0.000 0.000 0.000 
Var(Residual) 1.633 1.617 0.844 0.841 0.840 
Observations 544 544 544 544 544 


Note. The unconditional model reports the results of a cross-effect mixed model that controls fro respondent- and 
item-level variance components. Pooled results report the difference between treatment and control groups on raw 
item scores. Standardized results use the control group mean and standard deviation to center the raw scores. 
Standard errors in parentheses. * p < 0.05, ™ p< 0.01, *” p< 0.001 


How Facial Expressions Affect the Assessment of Instructional Actions 

We explore how facial expressions affects teachers’ assessment of instructional actions by 
estimating the effect of the inclusion of facial expressions on each individual PROSE item. 
Figure 3 visually presents the differences between the emoted and neutral conditions in their 
assessment of the instructional actions in each of the eight PROSE items. 

We note three main findings. First, in three items (a3022, a3012, a3101) the emoted 
scenarios received higher agreement with the normative actions than the neutral items. For the 
rest of the items, participants agreed less with the normative actions in the emoted items than in 
the neutral items. This finding suggests that facial expressions can both increase and decrease 
agreement with instructional actions. Second, we notice that all the 95% confidence intervals 
cross the red line at zero. This suggests that none of the differences that we observe are 
significant at the 95% level. This result likely indicates that our experiment was underpowered to 
detect significant differences in participants’ assessment on each individual item. Third, even if 
these differences are not significant, we can still make inferences about the direction of these 
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differences. That is, most of the confidence intervals for the negative point estimates are to the 
left of the dashed vertical line and most of the confidence intervals for the positive intervals are 
to the right of the dashed vertical line. This gives us evidence that the direction of the effect of 
emotions on participants’ assessment of instructional actions is in the same direction as the sign 
of the point estimate for most items. 


————_oTM@—1 oo 1 
e 
Emoted - 
e 
— 
! e 
t e 
se te —“ 
15 “1 -5 0 5 1 


®@ Item 3152 @ Item 1012 @ Item 3112 
@ Item 3162 ® Item 3032 ® Item 3022 
® Item 3012 ® Item 3102 


Note. The effects are measured on 6-point scales (unstandardized). 


Figure 2: Effect of Facial Expressions on Each PROSE Item 


Future Directions 

Together these two analyses suggest that: (1) the inclusion of facial expressions for 
representing emotions can impact the overall participant agreement with instructional actions and 
(2) the direction of that effect is not uniform—that is it can cause participants to agree with the 
actions of the teacher more or less favorably. With that, we suggest that more work will need to 
be done to understand exactly how the addition of emotions makes a difference for participants 
overall rating of an item. To begin this work, we examine some of the more qualitative 
differences between the items for which the addition of emotional cues had differing effects. To 
frame our quantitative results, a positive difference between the emoted and neutral items means 
that participants who saw emoted items deemed the teacher resolution of the teaching dilemma 
as more appropriate than the teacher’s normative (expected) action. Conversely, a negative 
difference between the emoted and neutral conditions means that the inclusion of emotions in the 
items made participants view less favorably the teacher’s follow-up action. 

With this in mind, we notice that the emotions in items where we have a negative difference 
are the direct result of the teacher’s initial action, either (1) the teacher ignoring or being 
insensitive to a student’s emotions (a1012, a3032, a3112) or (2) failing to anticipate students’ 
emotions (a3152, a3162). For these items we see participants casting judgment on the 
represented teacher’s actions. For example, after viewing a scenario in which a teacher overlooks 
students who are volunteering and cold calls a non-volunteering student with a confused 
expression, one participant reacted by saying “I prefer to acknowledge those who are willing to 
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contribute rather than embarrass those who are not. The students who did not raise their hands 
in this scenario look confused.” (emphasis ours, a3112, 4356). Similarly, after witnessing the 
teacher offer mementos to volunteers only to change her mind and draw sticks once students 
demonstrate excitement about the mementos, one participant commented “Teacher should have 
planned in advance to draw sticks, and told the students that's how they would determine who 
gets one” (emphasis ours, a3162, 5095). 

For the items for which we observed a positive difference, student emotions are a 
consequence of other students’ own actions and the represented teacher could be understood as 
resolving these situations (a3012, a3022, a3102). For these items, we see participants’ responses 
supporting the teacher’s actions, in spite of the students’ emotional state. For example, in the 
context of an item in which a student prefers to move on to other homework rather than help his 
peers and the teacher insists he help his peers, one respondent said, “This student is also being 
rude by being bold enough to state that they are sure they are correct. I have had students like 
this is[sic] the past and they have often benefited from checking answers when they thought they 
had done all their work correctly. The idea of working in groups is so that everyone can benefit.” 
(emphasis ours, a3102, 4571). These differences across the teachers’ reactions to emotions in the 
items makes us wonder whether the source of the emotional reaction plays a role in how teachers 
develop an agreement with instructional actions. We plan to explore this insight in future work. 


Discussion and Significance 

With the increase in interest in student non-cognitive outcomes (Battey & Neal, 2018), it has 
become more important to understand the role of emotions in teacher decision-making 
(Hargreaves, 1998). In this paper, we explored how emotions play a role in teacher decision- 
making. We found that facial expressions communicate emotions in scenarios of teaching and 
that teachers are responsive to these emotions when reacting to a scenario of teaching. While the 
effect size of the inclusion of emotions in PROSE items is small to moderate (a quarter of a 
standard deviation), this effect is non-negligible. Unaddressed emotions in items, either 
construed or represented, could bias participants’ assessment of instructional actions in ways that 
are difficult to predict a priori because facial expressions can both increase and decrease the 
direction of these assessments. 

Our previous work suggests these facial expressions perform as well as photographs of 
emotions (Dimmel et al., 2019) This highlights the affordances of the semiotic system 
(composed of eyebrows, eyes, and mouths) in enabling the reading of facial expressions while 
dramatically reducing the number of tokens used for such denotation. This can help in designing 
new instruments to assess teachers’ emotional competence. with the goal to support teacher 
professional development around the role of emotions in teaching. 

These findings have important implications for the use of cartoon-based scenarios of teaching 
in teacher professional development. We can use facial expressions to include the role of 
emotions in instructional decisions. Different teachers might interpret the emotions differently. 
This could help in describing the variation in instructional actions due to response to emotions. 
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